…#4280)

* Revert "Revert "Enable metrics using cloud-exporter" (#4208)"

This reverts commit 9978f40.

* rebase and add a missing change

---------

Co-authored-by: ashmeenkaur <ashmeen@google.com>

gcs/reader_count tracks the number of open and closed readers. ReadHandle label doesn't make sense there.

…xist while updating minObject (#4299)

- Ensuring MRDPool is not recreated if it doesn't exist while updating minObject.
- Making updateMRD no op for regional buckets.
- Making file UTs run for both zonal & non zonal bucket types.
- Updating fake bucket to handle append/overwrite.

Updating the retry logic in simple reader and adding some UTs

… log for better debugability (#4310)

* log max results and start offset in list log

* review comment

…ded (#4303)

Enable kernel reader for e2e tests except read_cache and buffered_read

…some fixes (#4309)

* add e2e tests for metadata prefetch and some fixes

* comment fix

* review comment

* fix list large dir tests so it can run with kernel list cache disabled

Adding additional UTs in kernel reader path.

* Remove redundant IsSetValue Interface and use Viper instead

…t error (#4315)

* rebasing with the latest

* Trigger Build

…e (1000) (#4323)

* feat(kernel-reader): setting 1000 as default value for inactive-mrd-cache

* fixing unit test

* test: add GCS metrics tests for request count, latencies, and download bytes.

* chore: add license header and use constant for read type metric attribute

* refactor: Rename o-direct flag to experimental-o-direct (#4155)

* Rename o-direct to experimental-o-direct

This change renames the hidden flag `o-direct` to `experimental-o-direct` to clearly indicate its experimental status. It keeps the flag hidden.

Changes include:
- Renaming `file-system.o-direct` to `file-system.experimental-o-direct` in `cfg/params.yaml`.
- Updating `cfg/config.go` to reflect the field rename to `ExperimentalODirect`.
- Updating usage in `internal/fs/fs.go`.
- Updating tests in `cmd/root_test.go` to use the new flag name.
- Marking the flag as experimental in the usage string.

* Rename o-direct to experimental-o-direct

This change renames the hidden flag `o-direct` to `experimental-o-direct` to clearly indicate its experimental status. It keeps the flag hidden.

Changes include:
- Renaming `file-system.o-direct` to `file-system.experimental-o-direct` in `cfg/params.yaml`.
- Updating `cfg/config.go` to reflect the field rename to `ExperimentalODirect`.
- Updating usage in `internal/fs/fs.go`.
- Updating tests in `cmd/root_test.go` to use the new flag name.
- Marking the flag as experimental in the usage string.

* Rename o-direct to experimental-o-direct

This change renames the hidden flag `o-direct` to `experimental-o-direct` to clearly indicate its experimental status. It keeps the flag hidden.

Changes include:
- Renaming `file-system.o-direct` to `file-system.experimental-o-direct` in `cfg/params.yaml`.
- Updating `cfg/config.go` to reflect the field rename to `ExperimentalODirect`.
- Updating usage in `internal/fs/fs.go`.
- Updating tests in `cmd/root_test.go` to use the new flag name.
- Marking the flag as experimental in the usage string.

* Rename o-direct to experimental-o-direct

This change renames the hidden flag `o-direct` to `experimental-o-direct` to clearly indicate its experimental status. It keeps the flag hidden.

Changes include:
- Renaming `file-system.o-direct` to `file-system.experimental-o-direct` in `cfg/params.yaml`.
- Updating `cfg/config.go` to reflect the field rename to `ExperimentalODirect`.
- Updating usage in `internal/fs/fs.go`.
- Updating tests in `cmd/root_test.go` to use the new flag name.
- Marking the flag as experimental in the usage string.

* Rename o-direct to experimental-o-direct

This change renames the hidden flag `o-direct` to `experimental-o-direct` to clearly indicate its experimental status. It keeps the flag hidden.

Changes include:
- Renaming `file-system.o-direct` to `file-system.experimental-o-direct` in `cfg/params.yaml`.
- Updating `cfg/config.go` to reflect the field rename to `ExperimentalODirect`.
- Updating usage in `internal/fs/fs.go`.
- Updating tests in `cmd/root_test.go` to use the new flag name.
- Marking the flag as experimental in the usage string.

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>

* feat: add GCS metrics tests for request count, latency, download bytes, and file cache interactions.

* feat: add GCS metrics tests for request count, latency, download bytes, and file cache interactions.

* fix: Corrected error return in fake bucket manager and refined GCS metrics test descriptions and setup.

* fix: Corrected error return in fake bucket manager and refined GCS metrics test descriptions and setup.

* fix: Corrected error return in fake bucket manager and refined GCS metrics test descriptions and setup.

* style: format gcs_metrics_test.go by adjusting blank lines.

* Empty commit

* chore: update copyright year to 2026

* Empty commit

* fix: Update GCS metrics test descriptions and increment request count verification

* fix: Update GCS metrics test descriptions and increment request count verification

* fix: Update GCS metrics test descriptions and increment request count verification

* fix: Update GCS metrics test descriptions and increment request count verification

* fix: Update GCS metrics test descriptions and increment request count verification

* fix: Update GCS metrics test descriptions and increment request count verification

* fix: Update GCS metrics test descriptions and increment request count verification

* fix: Add verification for request count in GCS metrics test

* feat: Add `gcs/read_bytes_count` metric verification and `ExperimentalParallelDownloadsDefaultOn` configuration to GCS metrics tests.

* test: validate gcs/retry_count metric for various retry categories

* style: adjust whitespace in test file.

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>

### Description
Move propagate context method to traceHandle.

### Link to the issue in case of a bug fix.
b/479039670

### Testing details
Existing unit tests & integration tests must pass.

### Any backward incompatible change? If so, please explain.
N/A

)

* Add metrics test for Kernel reader

…GKE env when kernel reader is enabled (#4316)

* write kernel config only for zonal buckets

* fix review comments

)

Pin dependencies to specific versions for compatibility.

…or consistency (#4314)

Refactors MrdSimpleReader to MrdKernelReader for better naming consistency with the enable-kernel-reader feature. The changes are mostly a rename and have been applied consistently across all relevant files.

Adding a trace log for short reads for better debuggability.

…o 3.11 (#4327)

* upgrade python to 3.11 for centos for cd pipeline

* correct commnt

Upgrading go sdk version to 1.59.2

* look up function change

* testing on Type cache deprecation changes

* adding unit tests

* small fix

* adding type cache deprecation arg in list req

* revert the default value of flag

* review comments

* rebase and fix tests

* nits

* review comments

* review comments

* add improved e2e cd script

Local improved e2e cd script, a better cleanup is req

faster run tial, commit to be reverted

testing with updated rhel conditions

Add header guard to prevent multiple sourcing

Rebase and add print statement for logs

* update header year

* PR reviews

* Update tools/cd_scripts/improved_e2e_test.sh

Added as suggested by gemini.

Co-authored-by: Mohit Kumar Yadav <mohitkyadav@google.com>

* pr reviews reduce local run if else conditions, add os type in details.txt for local runs

* remove default bucket, bucket argument created, need to add this in vm metadata

* remove comments

* newline at eof

---------

Co-authored-by: Mohit Kumar Yadav <mohitkyadav@google.com>

…red (#4332)

* remove version, add doc

* fix comment

### Description
Merge pending dependabot PRs.

Bumps [pytest](https://github.com/pytest-dev/pytest) from 7.3.1 to 9.0.3.  PR: [4615](#4615)

### Link to the issue in case of a bug fix.
b/509411478

### Testing details
1. Manual - NA
2. Unit tests - NA
3. Integration tests - NA
4. Performance test - NA (Updates to performance testing scripts themselves)

### Any backward incompatible change? If so, please explain.
NA

### Description
Adding streaming write fallback metrics that tracks the reason for the fallback and also which file mode & flag it fell back to staging writes. Useful for understanding why streaming writes fallback hasn't been picked .

### Link to the issue in case of a bug fix.
b/499889281

### Testing details
1. Manual 

Ran GCSFuse with streaming writes enabled.

Executed the command:



Existing File fallback test:

echo "test_data" > existing_file_test.txt on a new file.

Grepped the prometheus endpoint for the metrics using below: 

curl -s http://localhost:8080/metrics | grep "streaming_write_fallback"

No output for first time given that the file is new and streaming writes are allowed,
got output like below:

fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="existing_file"} 1

Executed the command multiple times in loops and given each time streaming writes cannot be used for the same reason of existing file, saw the below.

fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="existing_file"} 2

fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="existing_file"} 7

fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="existing_file"} 14


echo "test_data" > not_enabled_test.txt

got an output like below:

fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="not_enabled"} 2

Ran the following to test the concurrent limit breached.

for i in {1..10}; do    dd if=/dev/zero of=limit_breached_test_$i.txt bs=1M count=10 &  done; wait

Got the output for the metric fallback with appropriate reason.

fs_streaming_write_fallback_count{open_mode="write_only",write_fallback_reason="concurrent_limit_breached"} 7



similarly tested for not enabled and out_of_order_writes

2. Unit tests - added
3. Integration tests - NA

### Any backward incompatible change? If so, please explain.
N/A

…est migration] (#4543)

* update test_config.yaml

* setup file migration

* concurrent listing migration

* concurrent read migration

* update concurrent operations flags

* lint correction

* concurrent operations mounted tests passing

* correct create dir, and remove func for mountGCSFuse

* lint correction

* zonal flag addition

* gemini code review comment address

* remove log import

* lint correction and rebase

* lint correction for parallel tests

…uncated offsets (#4656)

…4664)

…ose for pirlo (#4668)

### Description
Remove metric mode log as it could potentially spam logs and not much actionable info could be derived from it and is getting logged even when metric is enabled, moreover such a log should be in the major product flow and logging it in metrics flow which is a side concern doesn't seem efficient.

### Link to the issue in case of a bug fix.
b/510655203

### Testing details
1. Manual - NA
2. Unit tests - NA
3. Integration tests - NA

### Any backward incompatible change? If so, please explain.
N/A

…for crcmod pip install (#4661)

* fix: Use requirements file with checksums for crcmod pip install

This addresses a security vulnerability where `pip install` was used
without manifest files and checksum verification. A new
`crcmod_requirements.txt` file was created with the hashes for the
`crcmod` package, and `os_utils.sh` was updated to use it with the
`--require-hashes` flag.

Co-authored-by: PranjalC100 <222632462+PranjalC100@users.noreply.github.com>

* add header

---------

Co-authored-by: google-labs-jules[bot] <161369871+google-labs-jules[bot]@users.noreply.github.com>
Co-authored-by: PranjalC100 <222632462+PranjalC100@users.noreply.github.com>

* dpchanges

* removed unused code

* change timeout to 15secs

* Update internal/storage/storage_handle.go

Co-authored-by: Mohit Kumar Yadav <mohitkyadav@google.com>

* changing multiplier to 2

* setting production retry config in createGrpcClientHandle method

* fixing crash

---------

Co-authored-by: Mohit Kumar Yadav <mohitkyadav@google.com>

* remove checkDirNotEmpty validation

* add fallback for error in deletion

* add non empty dest dir test

* assert error is ENOTEMPTY

… sleep bound (#4672)

* refactor: remove redundant prev state from ExponentialBackoff and update tests to use timing-based assertions

* fix review comments.

* fix loose test

* empty commit

* refactor: replace gcloud CLI call with direct metadata server curl request in package_gcsfuse.sh

* review comment: enable curl error reporting and failure on HTTP errors in package_gcsfuse.sh

…ce (#4662)

Re-runs go test with high counts to flag flaky tests. Pure read.

Signed-off-by: Arpit Jain <arpitjain099@gmail.com>

…y configs (#4687)

* feat: add validation checks and usage description updates for GCS retry configs

* refactor: remove TODO comment regarding non-negative value checks in NewRetryConfig

* fix review comments.

* review comment fixes: split validation test cases into separate success and error functions for readability

* review fixes: simplify the tests

* fix: allow >= 0s in max retry sleep as some tests use this value.

* trigger kokoro

… CD scripts (#4690)

* feat: add --release-package-bucket flag to allow configuration of the GCS release bucket

* feat: enforce required --release-package-bucket argument when not in local run mode

* Update tools/cd_scripts/improved_e2e_test.sh

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

---------

Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>

* parallelize in two kokoro vms- perf tests

* add child distributions

* correct the filenames as per kokoro configs

* testing purpose comment

* exit status and trap handlers

* Move distributed write benchmark check to the end of the block

* Enable write perf tests as well.

…in ExecuteWithRetries (#4691)

* feat: add support for custom should-retry predicates in retry execution functions

* empty commit

…it timer management in select blocks (#4692)

…4526)

* use migrated tests in improved_run_e2e script

* add to unmigrated pacakges

* rename variables

* reorder flags

* correction

* managed folder and rebase

* update variable replacement logic in packages

* key-file variable handling

…r Pirlo buckets (#4689)

* chore: upgrade Go toolchain to 1.26.3 and golang.org/x/net to v0.53.0 to fix CVEs

* refactor: introduce ShouldRetryWithoutLogging to decouple retry logic from logging and update retry executors accordingly

* refactor: remove unnecessary blank line in custom retry test file

* fix review comments.

* fix review comments.

* refactor: remove unnecessary whitespace in retry unit tests

* feat(skills): add dependency upgrade skill for GCSFuse

### Description
Add metric capture(handle or interface) methods for metadata_cache read count metric .

### Link to the issue in case of a bug fix.
b/512000913

### Testing details
1. Manual - NA
2. Unit tests - Added unittests cover testing the methods added for capturing the metrics.
3. Integration tests - NA

### Any backward incompatible change? If so, please explain.
N/A